Stereo Reconstruction 
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problem 




Goal: reconstruct the 3D shape of objects in the scene from 2 or more images. 



© Jorge Salvador Marques, 2003 



special case 




epipolar geometry 




epipolar line 

*2 = x 2 



© Jorge Salvador Marques, 2003 



depth 
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normalized disparity (df/b) 

depth estimates obtained from the true disparity value and from noisy disparity values 
with 0.001 errors. 



steps 



Stereo reconstruction involves 2 steps. 

matching: association of corresponding points in two (or more) images. 



triangulation: 
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calibrated cameras 



reconstruction 



problem: reconstruct X, from the projections x, x'and matrices P, P'. 



known cameras: P, P' 



ft-X P2X 

camera 1 : x. = — ~ x 9 = - 

P3-X P3-X 

camera 2: x-. ^ 1 ~ x 2 = — ~ 

p' 3 X p' 3 X 



I (frX)wX (p 3 X)x 2 =p 2 -X 
{ (p' 3 X)x\=p' v X (p' 3 -X)x' 2 =p' 2 -X 



X- homogeneous coord. 
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algebraic method 



(p 3 ■ X)xi = p 1 ■ X 
(p 3 -X)x 2 =P2-X 

(p } yX)x\=p\-X 

(p' 3 -X)x' 2 = p*2-X 

MX = 0 



4 equations 3 unknowns 



M 



T T 
Pi -P 3 x\ 

T T 
p 2 -p 3 x 2 

,r ,r , 

p l -p 3 x i 

,r ,r , 

_p 2 -p 3 x 2 



Least squares: X is a normalized eigenvector associated to the smallest eigenvalue. 

© Jorge Salvador Marques, 2003 



geometric method 




find the closest pair of points from both lines: 



optical rays 

X =C + at 
X'=C+a'f 



t = C-P*x 
f=C-F # x } 



[t-(t T ryf (c-c) 
i-(t T n 2 



[f-(t T f)tf(C-C) 



l-(t T t') 2 
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p# _ pT ^pp? ^ — 1 = P'T (p } P'T ) — ^ 



optical center 
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epipolar geometry 
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epipolar geometry 
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epipolar geometry (2) 

In general, two optical rays are non-coplanar and do not intersect. To guarantee that 
they meet the following condiction must hold 

r T Fx = o 

x,T- pair of corresponding points (homogeneous coord.) 
F -fundamental matrix 

The geometry of a pair of cameras is called epipolar geometry and it is 
characterized by F. 

The epipolar geometry is used in both steps of stereo reconstruction. 
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Lines in 2D 



A line 

can be expressed as follows using homogeneous coordinates 



l T x = 0 



There is a complete symmetry between points and lines in 2D using homogeneous 
coordinates. 



Intersection of two lines: 




line defined by two points: 



/ — Xj X X2 
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Matrix representation of cross product 



Cross product of two vectors 



axb ■ 



a 2 b 3 -a 3 b 2 
a 3 b x -a x b 3 
a l b 2 -a 2 b l 



Can be obtained by 



axb = [a] x b [a] x = 



0 - a 3 a 2 

a 3 0 —Oy 

-a 2 ay 0 



[a] x Is a skew symmetric matrix 
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Proof 




hypothesis: calibrated cameras (K=K'=I) and world 
coordinate system centered in camara 1 . 

p=[i o] f±[r I 



1 st step 

Define 2 points belonging to the optical 
ray defined by C and x 



X B = 



2nd step 

Project them by the 2 nd camera 

ro 



x' A =[R t] j =t x b =[R t] 
epipolar line 

l=t x7^=[t] x 7^ 
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Proof (2) 



3rd step 

3F belongs to the epipolar line 
x T l=() => X T [t] x Rx=0 

essential matrix 
When the cameras are not calibrated we can replace 
x^KT l x, x^K~ l Z 

and obtain 

x T Fx=0, F=K'- 7 [t] x RRr l 
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epipoles e epipolar lines 
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Propriedades da matriz fundamental 



• Fis a 3x3 matrix with rank 2 (det F=0) 

• fundamental property: 

• epipoles: 
•Epipolar lines: 
•linhas epipolares 

• relationship with P, P' 



x 1 Fx = 0 



Fe = 0, F T e'=() 



l = F T x\ l'= Fx 

r=F[k] x l k - line which does not contain the epipole 
F = [e']jFP # ) 
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Skew symmetric matrix 



0 -«3 ci2 
a 3 0 -a\ 
-«2 a\ 0 



uncalibrated cameras 



Reconstruction 



Matrix Fcan be obtained from pairs of corresponding points in both images. 
Fcan be estimated even if the projection matrices P, P are unknown 



Can we reconstruct the scene knowing matrix F? 
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Estimation of F 



F can be estimated from 8 or more pairs of points (7 are enough). 



Each pair of points defines a restriction 



U'n x' i2 l] 



fn fn fn 
hi hi fi3 
.hi hi h3. 



1 



Restrictions can be expressed in the form 
Mf = 0 



M : 



x' n x n x' n x l2 x\i x\ 2 x n x\ 2 x n x\ 2 x n x l2 1 



\_x ni x n i x nl x n2 x nl x n2 x nl x n2 x n2 x n2 x n2 x n2 
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constraints: //F//= 1 e det F =0 



8 point algorithm 



1 . normalization: move the mean to the origin and normalize variance 

2. Determine fundamental matrix 



• least squares: / is the eigenvector associated to the smallest 
eigenvalue of M T M; F is obtained from / 



• rank 2: p ■ 



UDV T 



where U, V are obtained by singular value 



decomposition of F and 



Denormalization 



F = TFT 



© Jorge Salvador Marques, 2003 



Theorem: the scene can be reconstructed from two uncalibrated cameras except for a 
projective transformation. 



The reconstructed points X^ are related to the true points Xj by 
X i = HX t 

Where H is an homography with 15 degrees of freedom. 



The reconstruction obtained in this way does not preserve angles, distances or 
parallel lines and it is denoted projective reconstruction 
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Reconstruction: uncalibrated cameras (2) 



Algorithm 

• estimate matrix F 

• define a pair of cameras compatible with F, e.g., 

P = [I 0] P'=[[e'] x F e'] 

• use standard stereo reconstruction methods to estimate X 
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Reconstruction: uncalibrated cameras (3) 



Metric reconstruction: 

Knowing pairs of corresponding points (X i9 Xf) 
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